ScreeNOT: Exact MSE-optimal singular value thresholding in correlated noise

We derive a formula for optimal hard thresholding of the singular value decomposition in presence correlated additive noise; although it nominally involves unobservables, we show how to apply even where noise covariance structure is not priori known or independently estimable. The proposed method, which call ScreeNOT, mathematically solid alternative Cattell’s ever-popular but vague scree plot heuristic from 1966. ScreeNOT has surprising oracle property: typically achieves exactly, large finite samples, lowest possible MSE matrix recovery, on each given problem instance, that is, specific threshold selects gives exactly smallest achievable loss among all choices noisy data set and unknown underlying true low rank model. method computationally efficient robust against perturbations structure. Our results depend assumption values have limiting empirical distribution compact support; this property, standard random theory, satisfied by many models exhibiting either cross-row correlation cross-column structure, also situations with more general, interelement Simulations demonstrate effectiveness at moderate sizes. paper supplemented ready-to-use software packages implementing algorithm: package Python (via PyPI) R CRAN).